-
-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
📊 Replace backported dataset by proper GHO #3171
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Quick links (staging server):
Login: chart-diff: ✅No charts for review.data-diff:= Dataset garden/who/2023-06-01/cholera
= Table cholera
~ Column cholera_case_fatality_rate (changed data)
~ Changed values: 2 / 2992 (0.07%)
country year cholera_case_fatality_rate - cholera_case_fatality_rate +
World 2016 1.831696 1.831765
Asia 2016 0.997723 0.997993
~ Column cholera_reported_cases (changed data)
~ Changed values: 3 / 2992 (0.10%)
country year cholera_reported_cases - cholera_reported_cases +
Iraq 2016 5 <NA>
World 2016 132118 132113
Asia 2016 18442 18437
2024-08-22 09:57:10 [error ] Traceback (most recent call last):
File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
return complexjson.loads(self.text, **kwargs)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads
return _default_decoder.decode(s)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode
obj, end = self.raw_decode(s)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/owid/etl/etl/datadiff.py", line 423, in cli
lines = future.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/owid/etl/etl/datadiff.py", line 416, in func
differ.summary()
File "/home/owid/etl/etl/datadiff.py", line 254, in summary
self._diff_tables(self.ds_a, self.ds_b, table_name)
File "/home/owid/etl/etl/datadiff.py", line 122, in _diff_tables
table_a = future_a.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
return copy(f, *args, **kw)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
do = self.iter(retry_state=retry_state)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
result = action(retry_state)
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/owid/etl/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
result = fn(*args, **kwargs)
File "/home/owid/etl/etl/datadiff.py", line 837, in get_table_with_retry
return ds[table_name]
File "/home/owid/etl/etl/datadiff.py", line 278, in __getitem__
return tables.load()
File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 312, in load
return self.iloc[0].load() # type: ignore
File "/home/owid/etl/lib/catalog/owid/catalog/catalogs.py", line 363, in load
return Table.read(uri)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 178, in read
table = cls.read_feather(path, **kwargs)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 350, in read_feather
cls._add_metadata(df, path, **kwargs)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 322, in _add_metadata
metadata = cls._read_metadata(path)
File "/home/owid/etl/lib/catalog/owid/catalog/tables.py", line 384, in _read_metadata
return cast(Dict[str, Any], requests.get(metadata_path).json())
File "/home/owid/etl/.venv/lib/python3.10/site-packages/requests/models.py", line 978, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
= Dataset garden/who/2024-02-14/gho_suicides
= Table gho_suicides_ratio
= Table gho_suicides
= Dataset garden/who/2024-05-20/vehicles
= Table vehicles
⚠ Found errors, create an issue please
Legend: +New ~Modified -Removed =Identical Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet Automatically updated datasets matching weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included Edited: 2024-08-22 09:57:13 UTC |
Marigold
force-pushed
the
cholera-refactor
branch
from
August 22, 2024 07:16
afb891c
to
ac9ca9b
Compare
Marigold
force-pushed
the
cholera-refactor
branch
from
August 22, 2024 09:34
ac9ca9b
to
50426e0
Compare
Marigold
force-pushed
the
cholera-refactor
branch
from
August 22, 2024 09:35
50426e0
to
7cbf66b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.